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ABSTRACT 

Emotional expression in Artificial Intelligence has gained lots of attention in recent years, people applied its 
affective computing not only in enhancing and realizing the interaction between computers and human, it also 
makes computer more humane. In this study, emotional expressions were applied into intelligent tutoring system, 
where learners’ emotional expression in learning process was observed in order to give an appropriate feedback. 
Emotional intelligent not only gives high flexibility to the interaction of tutoring system, it also to deepen its 
level of human interaction. 

This study uses dual-mode operation: facial expression recognition, and text semantics as the main elements in 
affective computing to understand users’ emotions. Text semantics are used to understand learner’s learning 
status, and the results would contribute to course management agents in order to choose the most appropriate 
teaching strategies and feedback to the users. Facial expression recognition allows interactive agents to provide 
users a complete sound and animation feedback.. 

INTRODUCTION 

Affective computing means to obtain facial expressions and signals of physical change aroused by emotions and 
feelings via various sensors and to recognize these signals so as to understand human emotions to give proper 
feedback (Li & Cheng, 2011; Li & Huang, 2007; Liao et al., 2010; Manovich, 2001). This is an emerging 
academic field, concerning the study of detecting human emotions and establishing proper emotional models so 
that emotions may be expressed in every possible way and even be transmitted on the Internet (MIT Media Lab, 
2008). In this case, affective computing is regarding two aspects: affection and emotion. Therefore, it will also 
need to detect information from both physical and psychological resources. According to Ammar et al. (2010), 
the latest scientific study has proved that emotions do exert great influence on decision making, perception, and 
learning. 

For the time being, most “intelligent tutoring systems (ITS)” place more emphasis on providing users with a 
highly flexible and interactive learning environment. Moreover, they may present users with proper learning 
materials along with teaching strategies based on their background knowledge. For example, when users do not 
reach desired grades, the tutoring system may lower the learning level timely to suit the users’ needs. 
Nevertheless, studies concerning the learning status of users are rare to be found. For instance, when a user’s 
learning capability is in decline, the cause for his/her weakened learning motivation may come from the 
individual’s mood swings rather than his/her poor capability of learning. In this case, it is hoped that the 
involvement of affective computing may help to observe users’ emotions and learning status so that the 
fluctuation may help the tutoring system to provide users with suitable courses and feedback (Lin et al., 2011a; 
Lin et al., 201 lb; Tsai et al., 2010). 

However, humans have a complicated way to express their emotions, such as facial expression, eye contact, 
body language, physiological phenomenon, and even words. In this case, any single method is not likely to 
obtain affection or situation in a complete manner. Therefore, this study suggests adopting facial expression 
recognition and text semantics as a dual-mode operation so that information regarding a user’s emotions and 
learning status may be discovered and understood better (Willems, 2011; Abulibdeh & Hassan, 2011; Yeo & 
Que, 2011). This study is aimed at applying affective computing to ITS so that a user’s learning status as well as 
immediate emotions may be considered an index for the reference of flexible tutoring courses. 
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LITERATURE REVIEW 

Kort et al.(2001) proposes a fundamental learning framework, whose abscissa means “Emotion Axis,” while the 
ordinate suggests “Learning Axis.” The farther an axis value on the right side of Emotion Axis is, the higher the 
positive energy will be and the further it on the left side is, the stronger the negative energy will be. On the other 
hand, the higher the “Constructive Learning” is, the stronger learning interests the use will have. Otherwise, the 
higher the “Un-Learning” is, the lower learning interests the user will have. 

It is made up of four models: interface pattern, expert pattern, student pattern, and tutor pattern (Koedinger and 
Corbett,2006). In recent years more and more tutoring systems focus on creating learners’ emotions in a tutoring 
environment, including emotional expression, empathy (Lester et al., 1999), and learners’ affection 
recognition(Conati et al., 2002). Besides, there are other studies concluding that the introduction of emotions to 
study is very likely to arouse users’ motivation. Emotions play a vital role in knowledge acquisition for humans 
(Vesterinen, 2001). They have been regarded as a considerably important factor to intrigue learners’ motivation, 
whereas motivation plays a vital role in knowledge acquisition (Mao & Li, 2010). In this case, “Affective 
Tutoring Systems (ATS)” is developed to detect students’ situations of learning and affection so as to give them 
timely emotional feedback and to correct their emotions of learning (Mao and Li, 2010). ATS is, in fact, 
designed based on the concept of ITS, aimed at mimicking real human reaction so as to adjust to students’ 
affection and situation in an effective manner(Sarrafzadeh et al., 2003; Sarrafzadeh et al., 2004; Vicente, 2003). 
According to Gerald (2004), negative emotions will weaken the learning status, while positive ones do benefit 
learning achievement. Moreover, Ammar et al., (2010) argues that the establishment of facial expression may 
strengthen the affective tutoring system in terms of the relationship of ATS and users. This study has proved that 
affective computing is good for monitoring users’ behavior in the learning process (Ammar et al., 2010; Lin et 
al., 2011a; Lin et al., 201 lb; Tsai et al., 2010). 

In early days, emotion recognition used to focus on mainly recognizing face and speech. Speech recognition 
detects mostly the frequency and energy of voices so as to generate a set of rules to identify emotions by means 
of statistics or machine learning. As for facial expression recognition, a great number of experts and researchers 
have conducted many relevant studies in these years. Among of them, “Facial Action Coding System (FACS)” 
developed by Ekman et al. (1978) may be the most well-known. This system identifies dozens of “action units 
(AUs)” based on facial muscles. It adopts facial contour beforehand to detect the positions of face, eyebrows, 
mouth, and nose. These positions are then served as points spreading into a plane so as to form an expression of 
features. Based on the movement of these feature points, each point’s displacement vector is carefully caculated. 
With respect to the recognition part, the collected data is added to various detectors such as SVM, Neural 
Network, and HMM for facial expression identification or hypothesis developed in accordance with 
psychologists’ definition. 

“Active Shape Model (ASM)” was firstly proposed by T.F. Cootes in 1992. Its former idea was “Active Contour 
Model (ACM).” The main concept of ASM is to establish an active shape model by training images of similiar 
objects. By applying the iterative method to decreasing the difference between the shape of model and that of the 
targeting object, the shape model can finally match the shape of targeting ojbect by means of adjusting the 
parameters. In this way, the extraction of feature points will be more likely to be accurate and precise. 

Digital art is an emerging genre of arts. It comes from Technical art, integrating computers, network, and 
multi-media altogether to present various aspects (Liao, 2003). Lin et al. (2004) believes that artistic works shall 
be able to reach viewers’ inner feelings. In addition to the affection demonstrated by the work itself, viewers are 
supposed to have deeper thoughts (Lin, et al., 2012; Wang et al., 2011). In this case, this art course may include 
art history, art appreciation, art creation, and art criticism (Liao, 2003). Moreover, lecturers may try to integrate 
information technology into art courses and to overcome issues of network connection in order to increase their 
teaching performance on art and culture, improve their instruction, and exert the positive effects of education 
(Lu, 2009). 

AUTHOR ARTWORK 
Research Method 

This system is divided into two parts: (1) Course and Interactive mechanism; (2) Emotion and Learning Status 
Recognition mechanism. Moreover, the two main parts are divided into smaller modules seen as Figure 1. 
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Figure 1 Flowchart of Affective Tutoring System 


Ontology-based Semantic Learning Status Recognition System 

The OMCSNet is served as the database for semantic hypothesis. After generating a hypothesis concerning the 
user’s learning status value, this study applies a semi-automatic voting mechanism to obtain the final learning 
status. This data will then become an indication for system agent to choose courses for users. To develop an 
ontology-based semantic learning status recognition system, this study adopts the OMCSNet (Open Mind 
Common Sense corpus, a semantic Network, 2003), a creation developed by Massachusetts Institute of 
Technology (MIT) Media Lab, to work with the voting mechanism so that this study is able to create the 
OMCSNetKai to complete the task (Lin et al., 201 la; Lin et al., 201 lb; Tsai et al., 2010). 

Algorithm of Facial Expression Recognition 

This mechanism of facial expression recognition and learning status recognition is created in accordance with the 
Detected Emotion based on Active Shape Model (DEASM) developed by this study. DEASM is the creation 
generated by Active Shape Model Library (ASMLibrary 5.0) SDK and the algorithm proposed by Ammar 
(2010). DEASM is applied to extract information of instant images via the webcam so that this study may match 
each frame with the active shape models to obtain the coordinates of feature points needed for the algorithm. 
Afterwards, this study conducts the algorithm of facial expression recognition to recognize users’ emotions. 
After extracting the necessary targeting feature points, this study conducts the definition of six spans of facial 
feature points (Di) proposed by Ammar (2010) to go with the six fundamental human emotions and develops the 
algorithm of facial expression recognition. 

Intelligent Tutoring System 

This study is aimed at developing an ATS on the basis of ITS, capable of detecting users’ emotions and giving 
them proper feedback. ATS suggests that the system is able to recognize a user’s learning and emotional status 
so that it may give the user a timely emotional feedback to help him/her back on track (Mao and Li, 2010). 
Despite the fact that many researchers have attempted to apply ITS to teach all sorts of fields such as algebra, 
geometry, mathematics, physics, and computer programming, rarely does it be applied to teach art courses, or 
even digital art. The following part is concerning the introduction of how this study makes use of the ATS it 
develops to teach digital art course (Lin et al., 2011a; Lin et al., 201 lb; Tsai et al., 2010). 

The interactive user agent module is divided into two subordinate modules: sound feedback and animation 
feedback. Based on six fundamental emotions, agents may react to users’ emotions in various sounds and 
animation. For instance, if the user has an expression of sadness, the agent will comfort him/her by wearing a 
caring look and asking, “Are you alright?” In accordance to the six fundamental emotions defined by Ekman & 
Friesen (1978) plus the “Neutral” emotion, The snapshot of the system is shown in Figure 2. 
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Figure 2: A snapshot of the system 


System Design and Evaluation 

This study intends to establish a more humane interactive mechanism by recognizing users’ emotions so that 
learners may enjoy a more flexible process of knowledge acquisition. Therefore, this study designs its system 
and evaluation as follows: (1) concept model; (2) prototype design; (3) expert-based heuristic evaluation; (4) 
ATS—A combination of emotion recognition and feedback; (5) triangulation evaluation on the final system, 
including questionnaires, observation, and interviews. The flowchart of system design and evaluation is seen as 
Figure 3. 
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Figure 3 Flowchart of System Design and Evaluation 


Prototype Design 

Expert-based Heuristic Evaluation on the Prototype 

After the cognitive walkthroughs, the heuristic evaluation process was performed then. Heuristic evaluation 
(Nielson & Molich, 1990) (Nielson, 1994) was developed by Jakob Nielsen according to usability exploring 
rules. These rules were called heuristics and used to evaluate whether the elements which made up the user 
interface were based on these principles. In Nielsen’s research, it was proved that experts can usually check out 
around 75% usability problems and skilled experts were able to observe a lot of usability problems on their own. 
Also, based on Nielsen’s advice, there should be four to six experts. In the research, we had five experts. 

Each expert spent one to two hours examining the prototype at least twice. First, experts grasped the procedure 
of the whole interactive interface manipulation and gained some knowledge about the work. Then, experts 
checked the usability problems of the entire artwork. Finally, experts discussed their evaluation results together, 
prioritized the problems, and offered solutions to them. 

Expert-based Heuristic Evaluation on the Prototype 

In the part of system estimation, it mainly evaluated system usability of the users on the research. We utilized the 
well-known questionnaire System Usability Scale (SUS) to evaluate the system usability. The questionnaire was 
revised with help from experts with significant experiences in the related fields. A 5-point scale ranging from 1 
as strongly disagree to 5 as strongly agree was used for the measurement. The revised version of the SUS 
questionnaire is in table 1 (Brooke, 1986; Tullis, 2004; Lutes, 2006; Isman, 2010). The revision majorly focused 
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on making SUS more suitable to artwork evaluation. 

SUS is a questionnaire to estimate users’ subjective feelings of the system and further know their degrees of 
satisfaction. In the aspect of system usability evaluation, the SUS is an efficient, time-conserving, and 
labor-saving way of subjective estimate. At present, it is widely applied in the system usability. After users 
finishing answering ten questions, the scale offers a formula which transfers the subjective feelings of users into 
the objective data information for analysis. That is, the score of SUS is used to evaluate usability of the system. 
The range of estimate score is from 0 to 100. The higher the score is, the more useful the system is and the more 
easily users can interact with it. 


_Table 1: SUS Questionnaire_ 

_System Usability Scale_ 

1. I think that I would like to interact with this work frequently 

2. I find the work unnecessarily complex 

3. I suppose the work is easy to use 

4. I think that I would need the support of a technician to help me use this work 

5. I find the various functions in this work are well integrated 

6. I suppose there is too much inconsistency in this work 

7. I would imagine that most people may learn to use this work very quickly 

8. I find the work not very user-friendly 

9. I feel very confident while using the work 

10. I need to learn a lot of things before I can get used to this work 

Expert-based Heuristic Evaluation on the Prototype 

By making use of the QUAN + QUAL Model, a better understanding can be provided for the phenomenon of 
interest (Creswell, 2008; Creswell & Clark, 2007; Gay, et al., 2009). Triangulation Evaluation consists of 
questionnaires, observations, and interviews. The final questionnaire was composed of QUIS and MSLQ. 
“Questionnaire for User Interaction Satisfaction (QUIS)” is issued to discover the subjective satisfaction of 
participants toward the user interface. This measurement tool can be used to measure satisfaction of the entire 
system, and to measure specific interface factors, such as screen visibility, terminology and system messages, 
learning factors, as well as functionality of a system. On the other hand, this study issues the “Motivated 
Strategies for Learning Questionnaire (MSLQ)” to discover how participants’ learning motivation is stimulated 
after using ATS. MSLQ is developed by Printric et al. (1991) and divided into six aspects with 30 questions in 
total. They are 4 questions of inner motivation, 4 of outer motivation, 6 of work value, 4 of control belief, 8 of 
self-efficacy, and 4 of learning anxiety respectively. Moreover, this questionnaire enquires participants in the 
manner of the five-point Likert Scale. 

After the end of the experiment, this study conducted focus group interviews and used Grounded Theory (Strauss 
& Corbin, 1990) to code and log the findings. Since most of the interview content correlate with enhancing 
learning interest, learning incentives, learning motivation and learning effect, coding schema was developed 
around these dimensions to organize user feedback and views, and to understand which research issues will be 
satisfied by the system developed in this research. 

Usability Evaluation Results on the Prototype 

After conducting the SPSS reliability analysis, this study concludes that its SUS reaches a Cronbach’s value of 
0.792, exceeding the overall reliability value of 0.7. The overall mean of SUS is 75 and the standard deviation is 
10.64. On balance, the result is determined as skewed left mesokurtic (skewness=-.492, kurtosis=.789), 
indicating that the cluster with high scores account for a high percentage among all users. In this case, most users 
are highly sensitive to the system usability of ATS. Afterwards, this study creates a final questionnaire and 
processes all users’ answers into a data shown as Table 2, which indicating participants are satisfied with the 
usability of ATS 


Table 2 Statisitic Result of SUS Scores 


Number of 

i 

average 

median 

Maximum 

Minimum 

Standard 

i • j • 

20 

75.00 

78.75 

87.50 

50.00 

10.64 


Based on the statistic data, this study selects the first and second top scores appearing in this five points scale and 
puts these two percentages together. As it is seen in Table 3, the overall subjective feeling is 78.50%. In Ql, 
there are 65.0% of users willing to use such a learning system as ATS. Moreover, in Q2, 80.0% of users agree 
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that this system is not complicated. In Q3, there are 90.0% of users consider this system easy to use, while in Q4, 
40% of users think that they need extra assistance from the technicians to operate this system. In Q5, over 90% 
of users consider the system functions well-integrated. What’s more, in Q6, 80% of users do not think the system 
contains many contradictions. In Q7, there are 85% of users believing they do not need much time to learn how 
to use this system and in Q8, over 90% of users think of using the system as not difficult. In Q9, 85% of users 
have strong faith in operating this system, while in Q10, 80% of users believe that they need to have some 
background knowledge prior to using this learning system. Based on the mean of SUS, 75.0, as well as the 
question analysis from Q1 all the way through to Q10, it is concluded that this system has good usability. 

_ Table 3 Descriptive Statisitcs of SUS _ 


Mean, standard deviation, skewness, and kurtosis Question Percentage of the Five Point Scale! % J 

1 2 3 4 5 


Qi 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

Q8 

Q9 

Q10 

Overall 

3.80 

.696 

.292 

-.734 

0 

0 

35.0 

50.0 

15.0 

3.85 

.489 

-.442 

1.304 

0 

0 

20.0 

75.0 

5.0 

4.15 

.587 

-.004 

.178 

0 

0 

10.0 

65.0 

25.0 

3.30 

.923 

.214 

-.595 

0 

20.0 

40.0 

30.0 

10.0 

3.95 

.394 

-.531 

4.985 

0 

0 

10.0 

85.0 

5.0 

4.00 

.795 

-.699 

.807 

0 

5.0 

15.0 

55.0 

25.0 

4.30 

.733 

-.553 

-.834 

0 

0 

15.0 

40.0 

45.0 

4.35 

.813 

-1.42 

2.379 

0 

5.0 

5.0 

40.0 

50.0 

4.40 

.754 

-.851 

-.609 

0 

0 

15.0 

30.0 

55.0. 

3.90 

.852 

-.930 

1.012 

0 

10.0 

10.0 

60.0 

20.0 

4.00 

.703 

-.492 

.789 

0 

4.0 

17.50 

53.0 

25.50 


Analysis of Questionnaire for User Interaction Satisfaction (QUIS) 

In this study, the result of QUIS reliability analysis in terms of its six aspects is seen as Table 4. Among the six 
aspects, except the reliability value of “terminology and system information” being 0.535, which is less than 0.7, 
the rest all reaches the general desired value 0.7. 


Table 4 Reliability Analysis of User Interaction Satisfaction Aspect 


Overall User Feedback 

Screen Presentation 

Terminology and System 

Infnrmnti nn 

0.869 

0.799 

0.535 

Learning operation system 

System performance 

User interface usability 

0.719 

0.833 

0.757 


As it is shown in Table 5, the score of “user interface usability” is much lower than the mean. On the other hand, 
the reason why the score of “terminology and system information” is slightly lower than the mean is probably 
because the interface layout is not user-friendly enough. Besides, the interface contains more English 
explanation than Chinese. In this case, to students who are native Chinese speakers, they may have difficulty 
operating the system in a direct method. Nevertheless, the score of “learning operation system” and “system 
performance” is 6.17 and 6.40 respectively, indicating that this ATS is still acceptable to users. It is concluded 
that (1) the usability of ATS is good. 

With respect to “overall user feedback” and “screen presentation,” the score is 7.05 and 7.30 respectively. They 
have the best two scores, indicating that though the interface usability may not be the most satisfying element, 
screen presentation does improve this weakness. Therefore, users have positive feedback concerning the 
operation of ATS as a whole. Since the “overall user feedback” scores as high as 7.05, it is concluded that (2) 
users are satisfied with ATS. 

However, in the aspect of “overall user feedback,” the non-experienced group scores 7.11, higher than the 
experienced group scoring 6.89. This may be the result that the non-experienced group has never experienced 
such a learning system before; therefore, their first experience is totally new and interesting to them. That’s why 
the system arouses their positive feedback. Therefore, it is concluded that (3) the interaction of ATS is indeed 
attractive to users. Table 4-6 demonstrates the result of descriptive statistics and T-test analysis of QUIS aspects. 
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Table 5 Descriptive Statisitcs of QUIS 



Experienced(22 

people) 

Non-experienced( 18 
people) 

total(40people) 

P-value 

(double 

-tailed 

test) 

average 

Standard 

deviation 

average 

Standard 

deviation 

average 

Standard 

deviation 

Overall user 
feedback 

6.89 

1.01 

7.11 

0.49 

7.05 

1.03 

0.415 

Screen presentation 

7.19 

0.732 

7.43 

0.77 

7.30 

0.94 

0.324 

Terminology and 

6.51 

0.47 

5.97 

0.40 

6.27 

0.91 

0.000* 

Learning operation 
system 

6.50 

0.70 

5.76 

0.41 

6.17 

1.106 

0.000* 

** 

System performance 

6.34 

0.50 

6.47 

0.66 

6.40 

1.25 

0.481 

User interface 
usability 

5.88 

0.64 

5.30 

0.58 

5.62 

1.09 

0.005* 

* 

Average 

6.55 

0.68 

6.34 

0.55 

6.47 

1.05 

0.204 


(P-value suggests the comparison between the experienced group with the non-experienced group.) 
Notice: * means p<0.05, ** means p<0.01, and *** means p<0.001 


CONCLUSION 

This study is aimed at creating and developing a reliable process of design and evaluation plus the digital art 
course of Affective Tutoring System. Based on the study objectives and questions stated in the very beginning, 
this study has concluded the following conclusion after compiling the user feedback: 

(l)ATS is easy to use. Moreover, the interaction is outstanding. Therefore, the compatibility of affective 
computing in the tutoring system is considerably good. (2)When interacting with ATS, users do achieve their 
desired objectives. Therefore, users are highly satisfied with ATS.(3)Since users have strongly positive feedback 
with respect to the affective recognition and agent feedback of ATS, it is indicated that the interaction of ATS is 
attractive to users. (4)Since users are interested in this informative digital art course, it is suggested that ATS is 
helpful in terms of increasing users’ motivation in learning the digital art course. 

This study attempts to follow the process of design and evaluation to establish a tutoring system based on the 
affective recognition and to teach digital art course via this system. This system combines two various 
recognition methods, OMCSNetKai and DEASM, to identify the user’s emotional status while he/she is 
inputting words and to provide the user with a suitable course level as well as agent feedback. It is expected that 
users may maintain good emotional status in the process of learning so as to increase their learning motivation. 
Besides, this study adopts an evaluation method combining both quality and quantity altogether to help this study 
better understand the true feelings of users toward the system. Therefore, the concluded feedback may serve as a 
reference for the system feasibility. 
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